fix(data-fabric): MULTILINE_MAX list reads are a preview, not always a marker - #3066
Conversation
…a marker The skill guidance and the multiline-max eval both asserted that records list and records query always return a size marker for a MULTILINE_MAX field. The backend now has two preview forms and which one a tenant returns depends on its rollout state: content truncated to 10,000 characters with a "...[Truncated]" suffix, or the legacy HasValue=true Length=N marker, sometimes with a trailing hint. Encrypted fields always return HasValue=true Encrypted=true. Only the single-record read is guaranteed to return the whole value. The eval's judge rubric was the part that made this a failing test rather than a stale doc: it required the agent to show a size marker, and explicitly scored "truncation" as a wrong theory. An agent that correctly described a truncated preview was marked wrong. The rubric now accepts either form and fails point b only for a real misunderstanding, such as claiming list returns the complete value. Point c, the filter rejection, is unchanged and still correct. Guidance is written so it holds on any tenant rather than naming a ring, and tells the agent not to branch on which form it received. The write-back warning is strengthened, because a truncated preview looks like ordinary content, so echoing one into records update destroys the tail of the stored value far less visibly than a marker string did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @aayushuipath's task in 2m 53s —— View job PR Review: fix(data-fabric): MULTILINE_MAX list reads are a preview, not always a markerSummaryUpdates the MULTILINE_MAX contract across four reference docs and one eval task to reflect that Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | No frontmatter changes in this PR |
| E2E Tests | OK | Existing task updated; 7 criteria intact, rubric broadened not weakened |
| Skill Body | OK | No SKILL.md changes |
| References & Assets | OK | 4 reference docs updated consistently; all internal links resolve |
| Repo Hygiene | OK | No secrets, no cross-skill deps, changes scoped to data-fabric domain |
Issues for Manual Review
- Tenant rollout state. The PR author calls this out explicitly: the diagnosis assumes the eval tenant has the truncated-content rollout active. If the nightly failure was caused by something other than the judge scoring points a/b wrong, this PR fixes a genuine doc staleness but may not fix the redness. Pasting the judge reasoning from the failing run would settle it either way.
- Domain accuracy. The 10,000-character truncation threshold and
...[Truncated]suffix are stated as facts. These match the PR description's claim about the backend source, but I can't independently verify the exact cutoff or suffix format.
Conclusion
Clean PR. The contract change is well-reasoned, internally consistent across all four reference files, and the eval rubric is broadened to match reality without weakening the test. The only verification gap is whether the nightly failure actually manifests as the judge scoring a/b wrong on a truncated-content tenant — the author flagged this transparently. No changes requested.
- Read review criteria and project conventions
- Read full diff
- Read full changed files for context
- Check for broken links, cross-skill refs, secrets
- Post review
| Branch
|
Claude finished @aayushuipath's task in 2m 15s —— View job Coder-eval task lint (advisory)1 task YAML changed; verdicts: 0 Critical, 1 High, 0 Medium, 0 Low, 0 OK. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run❌ High — PR body explicitly states: "Not run locally. I could not execute Per-task lint
|
ninja-shreyash
left a comment
ninja-shreyash
left a comment
There was a problem hiding this comment.
pls attach a passing coder eval run in PR desc
|
Attached a passing run to the PR description, thanks for the nudge. Ran via the
|
3 participants
What
skill-datafabric-integration-multiline-maxhas been failing in the daily coding-agent run. The cause is the eval's own judge rubric, not the agent and not the skill's mechanics.The rubric encoded a contract the backend no longer has exclusively. It required the agent to show that
records listreturned a size marker, and it explicitly listed truncation as a wrong theory:MULTILINE_MAX list and query reads now return one of two preview forms, and which one depends on the tenant's rollout state:
...[Truncated]HasValue=true Length=N, sometimes with a trailing hintHasValue=true Encrypted=trueOnly the single-record read is guaranteed to return the whole value. So on a tenant where truncation is active, an agent that described the preview correctly was scored wrong on two of three points, and the task went red no matter how well it behaved.
How
records get, so a real regression is still caught. Point c, the filter rejection, is untouched and still accurate.records-query.md(canonical section, renamed to "Preview vs Full Content" with both referring anchors updated),entity-schema.md,data-fabric.mdRule 21, anduipath-coded-apps/references/sdk/data-fabric.md, which carried the same claim for the SDK.records updatedestroys the tail of the stored value far less visibly than a marker string did.Tests
success_criteriaare intact; every other Data Fabric task YAML still parses.#multiline_max-fields--marker-vs-full-contentanchors remain anywhere in the repo.Passing run on this branch (via the
Run Coder Evalworkflow,agent: codex, coder-eval 0.11.5, the same framework version the PR gates use):final_status: SUCCESSmeans every one of the 7success_criteriamet its pass threshold, including thellm_judge(weight 3.0, threshold 0.5) that was scoring a correct answer as wrong before this change.This also resolves the open question the PR was carrying: the redness really was the judge rubric. The task now passes with the updated rubric and the updated guidance together, on the same tenant the daily uses.
Related
Same contract corrected in the TypeScript SDK's own docs: UiPath/uipath-typescript#703.